Skip to content

Conversation

bshastry
Copy link
Contributor

@bshastry bshastry commented Oct 2, 2025

Enable blocktest to read filenames from stdin when no path argument is provided, matching the existing statetest behavior. This allows efficient batch processing of blockchain tests.

Usage:

  • Single file: evm blocktest
  • Batch mode: find tests/ -name "*.json" | evm blocktest

🤖 Generated with Claude Code

Enable blocktest to read filenames from stdin when no path argument
is provided, matching the existing statetest behavior. This allows
efficient batch processing of blockchain tests.

Usage:
- Single file: evm blocktest <path>
- Batch mode: find tests/ -name "*.json" | evm blocktest

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@MariusVanDerWijden
Copy link
Member

The lint is red @bshastry

@bshastry bshastry force-pushed the cmd/evm-blocktest-stdin branch 2 times, most recently from 9d2a754 to 0e8268a Compare October 6, 2025 09:14
@bshastry bshastry force-pushed the cmd/evm-blocktest-stdin branch from 0e8268a to 94b4de0 Compare October 6, 2025 09:17
@MariusVanDerWijden
Copy link
Member

Been looking a bit into this. Its seems the whole concept of EndMarker should be removed. What I guess you want is to write out the results between the traces not only at the end of the test, right? This should be possible with the traceResult struct. I will push a commit on top which changes this, we can remove it again if you don't agree

@bshastry
Copy link
Contributor Author

bshastry commented Oct 6, 2025

Been looking a bit into this. Its seems the whole concept of EndMarker should be removed. What I guess you want is to write out the results between the traces not only at the end of the test, right? This should be possible with the traceResult struct. I will push a commit on top which changes this, we can remove it again if you don't agree

Yeah, basically I want a read protocol for jsonl that signals when blocktest results are complete, so I can break the jsonl scanner loop.

trace line 1 trace line 2

...
<- Scanner breaks on seeing end marker

Copy link
Member

@MariusVanDerWijden MariusVanDerWijden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This commit addresses all feedback from @rjl493456442:

1. Result printing location: Removed duplicate report() call from
   inside test loop. Now uses dedicated traceEndMarker on stderr
   instead of printing full results mid-execution.

2. Fork assignment: Fork now always assigned regardless of tracer
   or pass/fail status. Root only assigned when test succeeds,
   following correct semantics.

3. Log suppression: Removed automatic log suppression. Users can
   control logging via standard --verbosity and --log.file flags.
   Only one rare INFO log exists in blocktest path.

The traceEndMarker provides clear delimiter for trace parsers
(e.g., goevmlab) without duplicating test results. Format:
{"testEnd":{"name":"...","pass":true,"fork":"...","root":"..."}}

This follows precedent from Nethermind PR ethereum#9432.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
Copy link

@rv64m rv64m left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Code Review Summary

The review focused on adding stdin support to the evm blocktest command, aligning its functionality with statetest for improved batch processing. The implementation is straightforward and achieves its goal. The primary risk identified is code duplication and a minor behavioral inconsistency, for which a clear refactoring suggestion is provided to create a unified file collection utility. Overall, the change is a valuable, low-risk enhancement to the developer toolkit.

📋 Reviewed Changes

  • cmd/evm/blockrunner.go: Modified to read test file paths from stdin when no command-line arguments are provided.
  • cmd/evm/main.go: Contains existing utility functions like collectFiles which are related to the new logic.
  • tests/block_test_util.go: Test utilities, reviewed for context and potential impact from the changes.

📊 Architecture & Flow Diagrams

Diagram 1

graph TD
    A["evm blocktest"] --> B{"Args provided"}
    B -->|Yes| C["Read paths from args"]
    B -->|No| D["Read paths from stdin"]
    C --> E["Expand directories  collect files"]
    D --> E
    E --> F["Run tests on file paths"]
Loading

🤔 Review Quality Assessment

The review demonstrates comprehensive coverage, correctly identifying code duplication and the opportunity for a shared utility. The synthesis process refined this insight into a more holistic and actionable recommendation that includes refactoring statetest for consistency and addresses subtle behavioral differences. Confidence in the final, consolidated suggestions is high, as they significantly improve the long-term maintainability and robustness of the CLI tool.

💡 Suggestions Summary

  • Inline Comments: 0 suggestions on modified code lines
  • General Suggestions: 2 suggestions for overall improvements

📝 Additional Suggestions

The following suggestions apply to code outside the current PR diff:

1. 🔴 To improve maintainability and ensure consistent behavior across commands, refactor the file path collection logic into a shared utility function in cmd/evm/main.go. This new function, collectPathsFromInput, should handle reading from both command-line arguments and stdin, and also manage directory expansion for all inputs. This eliminates code duplication between blocktest and statetest and ensures consistent behavior.

File: cmd/evm/main.go (Line 360)
Confidence: 95%

// collectPathsFromInput reads paths from command-line arguments or stdin and expands them.
func collectPathsFromInput(ctx *cli.Context) []string {
	var paths []string
	if ctx.NArg() == 0 {
		// Read from stdin if no arguments are provided.
		scanner := bufio.NewScanner(os.Stdin)
		for scanner.Scan() {
			paths = append(paths, scanner.Text())
		}
	} else {
		// Read from command-line arguments.
		paths = ctx.Args().Slice()
	}

	// Expand all paths, whether from stdin or args, to discover test files.
	var expandedPaths []string
	for _, path := range paths {
		expandedPaths = append(expandedPaths, collectFiles(path)...)
	}
	return expandedPaths
}

2. 🔴 Update blockTestCmd to use the new collectPathsFromInput helper function. This simplifies the command's implementation, removes the duplicated logic, and ensures it benefits from the centralized, consistent method for discovering test files.

File: cmd/evm/blockrunner.go (Line 158)
Confidence: 95%

		paths := collectPathsFromInput(ctx)
		if len(paths) == 0 {
			return errors.New("no test files found")
		}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants